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OBJECTIVE 

This  Technical  Document  presents  a  plan  for  the  implementation  of  testability  into  military 
equipment  The  plan  advocates  the  generation  and  modification  of  official  MIL-type  documents  in 
order  (1)  to  establish  testability  as  a  design  discipline  and  (2)  to  result  in  a  family  of  testability 
documents  that  can  be  applied  when  procuring  military  hardware.  This  plan  consists  of  specific 
recommendations  in  six  functional  areas  comprising  testability. 


APPROACH 

The  task  was  accomplished  in  two  parts.  The  first  part  entailed  grouping  existing  and  proposed 
military  standards  and  design  guidance  notebooks  that  relate  to  testability  under  the  following 
testability  headings: 

•  Management 

•  Technical  Requirements 

•  Design  Guide 

•  Documentation 

•  Validation 

•  Interfaces 

The  second  part  was  accomplished  by  analyzing  these  groups  of  documents,  noting  deficiencies 
and  making  recommendations  to  alleviate  these  deficiencies.  In  addition,  a  survey  of  other 
agencies  and  commercial  companies  was  performed  to  determine  if  progress  toward  the  objective 
was  being  made  elsewhere. 


RESULTS 

Specific  recommendations  were  made  in  each  functional  area  These  recommendations  as  they 
apply  to  existing  and  proposed  MIDdocuments  are  summarized  below: 

Proposed  New  Documents: 

Military  Standards 

MIL- STD- ABC  Testability  Program  Requirements 

MII-STD-DEF  Testability  Demonstration 
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MIL-STD-GHI 


Testability  Analysis  and  Report 
MIL-STD-XXXX  System  Level  Performance  Monitoring 

Design  Guides 
Design  for  Testability  Guide 
Performance  Monitoring  Design  Guide 
Modification  -  Proposed  to  Existing  Documentation: 

Military  Specifications 
MILE- 16400 
MILT-24309 
MIL-T-28800 

Other  general  equipment  specifications  based  on  MIL- STD-454 
Military  Standards 
MILSTD-415 
MIL  STD-454 
MIL  STD-490 
MILSTD1521 
MILSTD2076 
MILSTD2077 
MILSTD211 1 
Cancellation  -  Proposed 
MILSTD1326 
MILSTD 1 345 
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MILSTD1519 


CONCLUSION 


The  plan  presented  herein  provides  the  necessary  framework  for  a  program  to  develop  these 
documents  as  well  as  for  the  definition  of  their  content,  organization  and  relationships.  This 
framework  is  sufficient  for  the  structuring  of  specific  tasks  in  each  functional  area  in  order  to 
achieve  highly  testable  design  in  military  equipment 

NOTE:  Testability  is  a  design  characteristic  which  allows  the  status  (operable, 

inoperable,  or  degraded)  of  a  unit  (system,  subsystem,  module  or  component) 
to  be  confidently  determined  in  a  timely  fashion. 
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1.0  INTRODUCTION 


The  requirement  to  treat  testability  as  an  independent  valid  design  objective  within  the  scope 
of  manual  and  automatic  testing,  has  been  recognized  only  recently.  In  the  past,  testability  was 
documented  by  inference  within  other  areas  such  as  maintainability  and  reliability.  As  a  result, 
many  varied  testability-related  documents  currently  are  in  print,  in  use,  and  being  proposed.  These 
documents  have  been  developed  over  a  period  of  many  years  and  are  generally  specialized  for  the 
individual  needs  of  the  varied  users.  This  results  in  a  number  of  differing  methods  for  achieving 
testability  in  hardware.  In  some  cases  these  documents  are  also  limited  by  the  state  of  the  tech¬ 
nology  at  the  time  they  were  written.  Additionally,  varying  degrees  of  conflict,  overlap,  and  dupli¬ 
cation  exist  between  these  documents.  It  is  therefore  expedient  for  the  testing  technology 
community  to  review  the  present  state  of  testability-related  documentation  to  determine  the  best 
approach  for  using  existing  documents  and  for  creating  new  ones. 

This  report  presents  a  “systems  approach”  to  the  review  of  testability-related  documents.  It 
also  presents  the  information  which  discloses  the  level  of  conflict,  overlap,  and  duplication 
between  current  testability-related  documentation.  This  includes  segmentation  of  the  subject  into 
major  areas  of  interest  Areas  that  may  require  changes  or  the  writing  of  new  documents  are 
identified  and  appropriate  recommendations  are  given.  The  end  result  is  a  guide  for  the  develop¬ 
ment  of  a  family  of  testability  documents  that  can  be  applied  during  the  procurement  phase  of 
shipboard  hardware,  and  will  aid  in  the  achievement  of  a  high  degree  of  testability  and  self-test 
features  in  the  hardware. 


2.0  BACKGROUND 

The  Testing  Technology  Office  at  the  Naval  Ocean  Systems  Center  (NOSC,  Code  921)  is 
currently  investigating  the  need  for  revisions  to  military  standards  and  other  documents  that  affect 
testing  and  performance  monitoring.  This  need  has  been  identified  as  arising  from: 

•  Testability,  as  treated  in  existing  documentation,  is  neither  recognized  as  a  design 
discipline  in  its  own  right,  nor  is  it  explicitly  mandated; 

•  Program  managers  having  cognizance  over  equipment  development  efforts,  have  no  single 
cohesive  thread  of  documentation  from  which  to  evoke  this  design  discipline  in  their 
programs; 

•  Some  existing  documentation  does  not  reflect  the  latest  technology, 

•  Potential  areas  of  conflict  or  incompatibility  exist  between  existing  standards  and  guide¬ 
lines,  as  well  as  those  being  prepared  in  response  to  new  requirements. 

In  the  course  of  the  investigation,  it  was  apparent  that  the  application  of  a  “systems  approach” 
to  the  organization  of  these  standards  would  prove  the  most  effective  method  of  resolving  these 
issues.  The  term  “systems  approach”  simply  means  to  scope  out  and  define  the  entire  problem 
prior  to  the  definition  of  the  solution  and  to  maintain  this  viewpoint  when  delimiting  each  piece  of 
the  solution.  This  approach  resulted  in  the  present  definition  for  subsequent  generation  of  a 
family  of  testability-related  documents  for  use  by  Testability  Engineers  as  well  as  Program  and 
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Acquisition  Managers.  This  family  of  documents  will  allow  the  establishment  of  testability  as  a 
design  discipline  at  the  same  level  as  the  requirements  for  performance,  reliability,  maintainability, 
etc.  The  development  of  shipboard  hardware  during  the  R&D  phase  is  a  mix  or  compromise 
between  the  various  engineering  disciplines.  In  the  past,  the  engineering  discipline  of  maintain¬ 
ability  has  been  the  primary  method  of  implementing  testability  into  the  hardware.  The  primary 
proof  of  testability  has  been  the  maintainability  demonstration  test  which  occurs  during 
qualification  testing.  The  main  thrust  of  this  test  is  aimed  more  at  the  mean-time-to-repair 
(MTTR)  rather  than  the  testability  of  the  hardware.  The  MTTR  figure  can  be  met  using  a 
relatively  untestable  hardware  design  by  relying  on  a  sufficiently  complex  test  scheme.  Testability 
impacts  the  efficiency  of  testing  as  well  as  the  MTTR 

Establishing  testability  as  a  design  discipline  will  place  it  at  an  increased  attention  level  when 
the  cost  and  engineering  trade-offs  are  performed  during  the  R&D  phase  of  the  hardware  develop¬ 
ment  This  will  increase  the  testability  of  military  hardware  because  it  is  more  cost  effective  and 
practical  to  design  testability  into  the  prime  equipment  than  to  develop  elaborate  test  schemes  to 
overcome  design  deficiencies. 


3.0  APPROACH 

A  two  phased  approach  was  taken.  In  the  first  phase,  a  survey  was  conducted  over  a  wid^ 
range  of  military  standards,  specifications  and  design  guidance  notebooks.  A  total  of  23  existing 
and  proposed  documents  were  included  on  the  basis  of  their  potential  impact  on  or  association 
with  testability.  Each  document  was  analyzed  in  the  following  categories:  . 

•  Major  topics 

•  Level  of  detail 

•  Area  of  coverage 

•  Constraints 

•  Overlap  between  documents 

•  Contributions  toward  testability 

Some  of  the  documents  surveyed  specify  requirements  in  the  related  fields  of  maintainability 
and  reliability.  They  were  included  because  they  have  potential  impact  on  testability.  However, 
they  also  can  serve  in  an  instructional  capacity  due  to  the  fact  that  they  support  relatively  mature 
but  related  design  disciplines.  Thus,  the  previous  experience  and  example  expressed  in  the 
maintainability  and  reliability  documents  could  be  used  in  this  effort  Specifically,  the  relationship 
of  the  general  equipment  standards  and  specifications  was  examined  with  respect  to  reliability, 
maintainability  and  related  fields. 

The  second  phase  was  based  on  hardware  procurement  considerations.  From  the  aspect  of 
procurement  it  was  found  that  each  document  or  group  of  documents  support  one  or  more 
functions  as  follows: 
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•  An  overall  management  document  for  conducting  a  testability  program; 

•  A  group  of  documents  that  addresses  the  testability  technical  requirements  for  hardware; 

•  A  group  of  documents,  such  as  a  Design  Guide,  which  aid  the  contractor  in  implementing 
testability  into  the  hardware; 

•  A  group  of  documents  that  specifies  how  the  testability  data  is  to  be  prepared  and 
presented  (Design  Review  Data,  Technical  Requirements  Data  [TRD],  Test  Programs 
Sets  [TPS],  etc.); 

•  A  document  that  specifies  the  methods  for  the  validation  of  testability  (demonstration 
tests); 

•  A  group  of  documents  that  specifies  the  interfaces,  data  transfer,  electrical  character¬ 
istics,  and  sensor  selection  in  support  of  test/monitoring. 

An  analysis  was  conducted  on  a  function  by  function  basis.  Final  conclusions  were  then 
collected  and  summarized  into  recommendations. 

Additionally,  a  survey  of  “other  agencies”  and  commercial  companies  was  performed  to 
ascertain  the  procurement  methods  being  used  to  obtain  testability  of  hardware.  This  was  to  get  an 
indication  of  the  present  state  of  testability  relative  to  current  procurement  practices.  The 
following  agencies  and  commercial  companies  were  surveyed: 

•  U.S.  Air  Force,  Air  Force  Logistics  Command,  MATE  Program; 

•  U.S.  Army  Communications  Research  and  Development  Command  (CORADCOM); 
Direct  Support  Automatic  Test  Support  System  (DS-ATSS)  Program; 

•  Boeing  Company,  Commercial  Airplane  Division; 

•  Lockheed,  Commercial  Airplane  Division; 

•  NASA. 

The  methods  currently  being  used  by  the  groups  surveyed  at  the  time  of  this  report  are  as 
follows: 

U.S.  Air  Force,  MATE  Program 

These  Modular  Automatic  Test  Equipment  (MATE)  Testability  guides  were  not  available  for 
review  because  they  are  competition  sensitive.  The  significant  factor  is  that  the  subcontractors  are 
presenting  their  methodology  for  testability  to  the  Air  Force,  as  opposed  to  the  Air  Force 
Logistics  Command  designating  the  methodology. 
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U.S.  Amy,  CORADCOM 


The  group  at  the  U.S.  Army  Communications  Research  and  Development  Command 
(CORADCOM)  do  not  have  testability  standards.  Instead,  they  have  taken  information  from  an 
independently  funded  testability  design  study  and  developed  a  testability  portion  of  a  hardware 
RFP  for  “Direct  Support  Automatic  Test  Support  System  (DS-ATSS)”.  At  the  time  of  this 
report,  the  submitted  proposals  had  not  been  evaluated.  Testability  requirements  were  placed  in 
the  performance  specification,  and  data  items  in  accordance  with  the  following  DIDs,  were 
requested.  Appendix  C  contains  these  DIDs  in  their  entirety. 

Item  DID 

Testability  Program  Plan  DI-A-XXX1 

Testability  Analysis  Report  DI-R-XXX2 

Test  Requirements  Document  DI-T-3734A 

Testability  Demonstration  Plan  DI-T-XXX4 

Except  for  the  Test  Requirements  Document,  the  data  items  are  new. 

Boeing 

To  achieve  testability  in  hardware,  the  Boeing  Commercial  Airplane  Division  utilizes  main¬ 
tainability  requirements,  implementation  of  these  requirements  by  the  hardware  subcontractor,  and 
design  reviews  and  testing.  They  do  not  have  testability-only  documents,  nor  do  they  treat 
testability  as  a  separate  discipline.  Testability  related  requirements  are  placed  in  the  procurement 
specification. 

Lockheed 

The  approach  utilized  by  Lockheed  is  similar  to  that  used  by  Boeing.  The  pertinent  factor 
being  they  do  not  have  a  specific  “Testability  Document”,  but  use  maintainability  requirements  to 
achieve  testability. 

NASA,  Space  Shuttle,  Orbiter 

NASA  has  taken  the  position  that  most  of  the  hardware  for  the  Orbiter  portion  of  the  Space 
Shuttle  will  be  repaired  by  the  individual  suppliers  and  testability  would  not  be  cost  effective. 

They  justify  this  since  most  of  the  hardware  is  low  volume  and  off-the-shelf. 

The  “Other  Agencies”  surveys  resulted  in  the  following  general  conclusions: 

•  Testability  requirements  as  a  design  discipline  are  not  generally  specified  in  current 
acquisition  documents; 
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•  Maintainability  requirements  continued  to  be  the  most  common  method  of  installing 
testability, 

•  The  maintainability  and  testability  requirements  are  placed  in  the  “Requirements”  section 
of  specifications.  The  inclusion  of  these  maintainability  and/or  testability  requirements 
by  the  procuring  activity  without  the  aid  of  any  associated  “Testability  Documents”  is 
prevalent 

NOTE:  An  exception  to  the  use  of  maintainability  as  a  method  of  achieving  testability 
is  being  attempted  by  the  U.S.  Army  CORADCOM,  DS-ATSS  program. 
However,  the  program  has  not  progressed  sufficiently  to  provide  any  concrete 
conclusions. 


4.0  RESULTS 

The  results  of  this  study  are  presented  by  testability  function.  These  testability  functions  are  as 
follows: 


Testability  Function 

Paragraph 

Management 

4.1 

Technical  Requirements 

4.2 

Design  Guide 

4.3 

Documentation 

4.4 

Validation 

4.5 

Interfaces 

4.6 

The  Testability  Matrix,  Figure  1,  depicts  the  testability  document  grouping  by  function  along 
with  deficiencies/recommendations  by  function. 


4.1  MANAGEMENT 

Testability  has  not  been  generally  treated  as  a  separate  discipline  and  therefore,  except  for  the 
proposed  MID  STD- XXX  (W.  Keiner)  “Testability  Acquisition  Program  Requirements  for 
Electronic  Equipments  and  Systems”,  there  are  no  documents  solely  dedicated  to  the  overall 
management  and  validation  of  testability.  Testability  as  a  design  discipline  is  a  management 
function.  The  function  of  testability  management  during  the  procurement  phase  of  development 
requires  a  MID  STD  similar  to  MIDSTD-470  (maintainability)  and  MIDSTD-785  (for  reliability). 
The  document  “A  Study  of  Testability  Standardization  for  Electronic  Systems  and  Equipment” 
(W.  Keiner),  contains  the  testability  management  elements  with  which  to  develop  a  testability 
management  document,  MIDSTD-ABC  (Proposed).  “Testability  Acquisition  requirements  for 
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DESIGN  GUIDE 


MIL-STD-ABC  (Proposed) 


MIL- STD- XXX  (Proposed)  W.  Kemer 

MIL-E-16400 

Mll-T-28800 

MIL- STD-454 

MIL- STD-41  5 


Built-In  Test  (BIT)  Design  Guide 
Operational  Monitoring  (ORMS)  Design 
Guide  (Proposed) 

Design  for  Testability  Guide.  W  Kemer 
( Proposed) 

MIL- STD- 1390 
MIL- STD-1  364 


MIL- STD-1 51  9 
MIL- STD-1  345 
MIL- STD-2076 
MIL- STD-2077 
D-790 

MIL-T-24309 
MIL- STD-21  1  1 
MIL- STD-24255 
MIL- STD-490 
M1L-STD-GH1  (Proposed) 
MIL-T-24424 
MIL- STD-31  10 
MIL-STD-1  521 


MIL- STD-471 
MIL-STD-DEF 

Demonstration  (Proposed)  Testability 


MIL-STD-XXXX  (Proposed)  NOSC) 

MIL-STD-1 657 
SDMS  (I/O  Modules) 

MIL-STD-1 553 
MIL-STD-1 397 
DO  D- STD-1 399 
IEEE-STD-488 

MIL-STD-X.  On-Line/Off-Line 
Interface  (Proposed) 

MIL-STD-1 326 


Testability  Program 


Testability  Reqmts 
Equipment  Spec. 
Equipment  Spec. 
General  STD. 
Testability  Reqmts. 


Released  Design  Guide 
(Proposed) 


A  document  does  not  exist  in  Testability  as  a  discipline  Need  an  operational  moni- 
released  form  to  manage  a  does  not  exist  in  equipment  toring  and  design  for  test- 

testability  program  by  an  specs  or  general  standards.  ability  guide  to  bridge 

R&D  contractor.  educational  gap. 


Utilize  W.  Keiner  study  to  Change  the  equipment  Draft  an  operational  moni- 

develop  a  testability  program  specifications  and  MIL-STD’s  toring  design  guide.  Convert 

document  along  with  sup-  to  include  testability  as  a  the  W.  Keiner  testability 

porting  DID'a.  design  discipline.  course  to  a  testability  design 

guide. 


Figure  1.  Testability  Matrix 


DESIGN  GUIDE 


DOCUMENTATION 


VALIDATION 


INTERFACES 


Released  Design  Guide 
Proposed) 

i  Proposed) 

Level  of  Repair 
GPETE 


TRD 

TRD 

TRD 

TPS 

TPS 

TSP 

TRS 

TRS 

Specification  Practices 
Test.  Anal.  Et  Report 
TRS 
ROR 

DSN  Reviews/ Audits 


Maintainability  Test 
Testability  Demonstration 
(Proposed) 


System  Level  Monitoring 
( Proposed) 

Physical  Interface 
Equipment  Spec. 

Series  Data  Bus 
Computer  Interfaces 
Shipboard  Interface 
Parallel  Data  Bus 
(Proposed) 


Need  an  operational  moni¬ 
toring  and  design  for  test¬ 
ability  guide  to  bridge 
educational  gap. 


Testability  analysis  docu¬ 
ment  not  released.  Duplica¬ 
tion  in  TRD  area.  Testability 
test  document  not  released. 
TRD  and  TPS  documents  not 
universal. 


MIL-STD-471  does  not  place 
priority  on  how  well  the  test 
equipment  works,  but  how 
long  it  operates. 


On-Line  Testability 


Interfaces  for  shipboard  hard¬ 
ware,  both  on-line  and  off¬ 
line  have  not  been 
standardized. 


Draft  an  operational  moni¬ 
toring  design  guide.  Convert 
the  W.  Keiner  testability 
course  to  a  testability  design 
guide. 


Convert  a  portion  of  W. 
Keiner  study  to  a  testability 
analysis  document.  Develop 
a  universal  TRD  and  TPS 
document  utilizing  MIL- 
STD's  2076  and  2077  as  a 
base. 


Convert  a  portion  of  the 
W.  Keiner  study  to  a  testabil¬ 
ity  MIL-STD  for  validation. 


Standardize  the  interface 
for  both  on-line  and  off-line 
shipboard  hardware. 


Figure  I .  Testability  Matrix 
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Electronic  Equipment  and  Systems"  (MIL  STD-XXX,  Proposed)  will  also  assist  greatly  in 
testability  management  The  proposed  allocation  of  technical  data  from  these  reports  is  presented 
in  the  Technical  Requirements  section  of  this  report  The  establishment  of  testability  as  a  design 
discipline  and  the  organizational  role  of  the  testability  engineer  are  examined  in  the  material  to 
follow. 


4.1.1  TESTABILITY  ENGINEERING 

The  discipline  of  testability  emerged  primarily  thro.igh  the  maintainability  function.  Therefore, 
consideration  must  be  given  to  maintainability  and  similar  disciplines  to  review  the  pitfalls  that 
accompanied  their  emergence  as  entities.  Similar  difficulties  for  testability  can  then  be  avoided. 

Historically,  in  engineering  companies,  maintainability  has  been  a  support  function,  not  part  of 
the  performance  design  engineering  team.  As  such,  it  has  been  staffed  under  an  engineering 
support  or  quality  assurance  group  along  with  reliability.  This  tends  to  give  the  discipline  a 
classical  watchdog  ( quality  assurance  role)  connotation  which  tends  to  form  a  barrier  between  the 
performance  design  engineering  team  and  the  maintainability/reliability  engineers.  The  maintain¬ 
ability/ reliability  functions  achieve  the  most  efficient  implementation  when  a  working  group 
relationship  is  established  with  the  performance  design  engineering  group. 

To  date,  the  implementation  of  testability  into  the  hardware  has  not  been  a  high  priority  item 
even  within  the  scope  of  a  maintainability  demonstration.  During  such  demonstrations,  the 
contractor  usually  supplies  (and  thus  controls)  the  list  of  faults  along  with  their  implementation, 
and  thus  assures  the  successful  passing  of  the  test  despite  the  presence  of  any  testability 
shortcomings.  Classically  the  implementation  of  testability  has  been  handled  by  assigning  the 
design  effort  to  the  same  engineer  that  had  the  design  responsibility  for  performance.  This  usually 
causes  a  priority  and  time  scheduling  problem  that  results  in  testability  being  relegated  to  a  “when 
time  permits”  situation.  Also  contributing  to  the  problem  is  the  fact  that  the  performance  design 
engineer  does  not  normally  have  the  experience  or  background  in  testability-related  disciplines 
(ATE,  ATLAS,  TPS,  etc.)  which  are  instrumental  in  achieving  a  testable  design. 

In  any  consideration  of  adding  the  testability  function  to  the  duties  of  maintenance  or 
reliability  engineers,  the  following  considerations  must  be  made: 

•  Maintainability  engineers  tend  to  be  involved  in  the  “packaging”  of  the  hardware  by 
function,  performing  maintainability  predictions,  and  in  the  maintainability  demonstration 
tests,  along  with  monitoring  the  design; 

•  The  reliability  engineer  is  involved  with  parts  procurement,  parts  application,  reliability 
prediction,  failures,  and  the  reliability  demonstration  tests,  along  with  monitoring  the 
design; 

•  Because  of  the  function  of  testability,  (i.e.,  the  confident  timely  determination  of 
equipment  status),  the  testability  engineer  must  be  more  hardware  design  oriented  than 
the  maintainability  and  reliability  engineer. 


Due  to  the  differences  in  the  design  disciplines  of  testability  as  compared  with  maintainability  and 
reliability,  the  testability  function  is  optimized  by  assigning  the  testability  engineer  as  part  of  the 
performance  design  engineering  team.  The  enhancement  of  the  working  relationship  between  the 
testability  engineer  and  performance  design  engineers  should  increase  the  application  of  testability 
into  the  design  process. 


4.1.2  TESTABILITY  ENGINEERING  EFFECTIVENESS 

Effective  development  of  hardware  during  the  design  phase  is  a  composite  of  all  engineering 
disciplines  which  results  in  a  cost-effective  design.  All  design  disciplines  (including  testability)  have 
to  be  designed  in,  not  tested  in  after  the  fact  The  establishment  of  testability  as  a  design  discipline 
through  testability  requirements,  Contract  Data  Requirements  List  (CDRL)  items,  and  a 
testability  demonstration  test  will  result  in  more  testable  hardware. 

The  effectiveness  of  the  testability  engineering  role  will  depend  on  the  following: 

•  A  close  working  relationship  with  the  performance  design  engineering  team; 

•  A  testability  demonstration  test  as  part  of  the  qualification  tests.  This  places  importance 
on  testability  as  a  design  discipline,  an  attention  getter.  Testability  test  results  should  also 
be  considered  as  Design  Review,  Audit  (FCA/PCA)  requirements; 

•  A  clearly  defined  testability  requirement  in  the  performance  specification.  This  avoids 
ambiguities  when  design  trade-offs  are  performed  which  affect  performance,  testability, 
maintainability,  reliability,  safety,  etc.; 

•  Status  as  a  design  discipline  equal  to  performance,  maintainability,  reliability,  safety,  etc. 
This  is  achieved  by  program  requirements  which  are  attention  getters  for  the  testability 
discipline.  These  program  requirements  are  as  follows: 

-  Testability  Program  Plan; 

-  Testability  Analysis; 

-  A  subject  for  the  Preliminary  Design  Review  (PDR)  and  Critical  Design  Review  (CDR) 
as  specified  in  the  performance  specification,  proposed  testability  requirements, 
MIL-STDs  and  CDRLs; 

-  Testability  demonstration  test  as  part  of  the  qualification  tests. 


4.2  TECHNICAL  REQUIREMENTS 

Technical  requirements  pertain  to  those  testability  conditions  that  are  specified  to  a  contractor 
for  the  purpose  of  delineating  desired  limits  on  the  hardware  design.  These  testability  requirements 
are  in  MIL-equipment  specifications,  MIL-STDs  and  the  hardware  performance  specification. 
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The  hardware  performance  specification  (per  MILSTD-490)  is  the  starting  place  for  the  analysis 
herein,  followed  by  the  MILequipment  specifications  and  testability  MIL- STDs. 

The  following  documents,  as  shown  in  Figure  1 ,  are  grouped  under  the  Technical  requirements 
function: 

•  MILSTD-XXX  Proposed,  W.  Keiner,  Testability  Acquisition  Program  Requirements 
for  Electronic  Equipments  and  Systems; 

•  MILE- 16400,  Electronic  Equipment,  Naval  Ship  and  Shore:  General  Specification; 

•  MILT-28800,  Test  Equipment  for  use  with  Electrical  and  Electronic  Equipment, 
General  Specifications  for, 

•  MILSTD-454,  Standard  General  Requirements  for  Electronic  Equipment; 

•  MILSTD-415,  Test  Provisions  for  Electronic  Systems  and  Associated  Equipment, 
Design  Criteria  for. 


4.2.1  HARDWARE  PERFORMANCE  SPECIFICATIONS 

These  hardware  performance  specifications  are  drafted  to  the  applicable  appendix  of  MIL 
STD-490,  “Specification  Practices".  This  standard  does  not  currently  make  provisions  for 
testability  as  a  specification  item,  as  it  does  for  maintainability  and  reliability.  MILSTD-490  is 
discussed  here  only  in  the  context  that  testability  requirements  are  placed  in  the  hardware  perfor¬ 
mance  specification.  MILSTD-490  is  grouped  in  the  Testability  Matrix,  Figure  1,  under  the 
documentation  function. 


4.2.2  APPLICATION  OF  REQUIREMENTS 

The  application  of  a  given  program  design  discipline  to  a  family  of  military  equipment 
specifications  is  dependent  upon  the  ability  to  pass  requirements  associated  with  that  particular 
discipline  from  general  documents  to  detailed  equipment  specifications.  The  requirement  of  main¬ 
tainability,  as  applicable  to  electronic  equipment  design,  is  used  in  the  following  example  since  it 
is  a  long  acknowledged  design  discipline. 


4.2.2. 1  Maintainability  Example 

General  specifications,  such  as  MILE- 16400  for  electronic  equipment  and  MILT-28800  for 
test  equipment,  specify  requirements  for  maintainability  (as  well  as  other  acknowledged  program 
disciplines).  These  requirements  are  consolidated  and  standardized  in  MILSTD-454,  “Standard 
General  Requirements  for  Electronic  Equipment".  Certain  of  these  requirements  complement 
each  other  in  application  to  equipment  design  and  can  be  categorized  as  hardware  and  program 
(design  discipline)  requirements  as  follows. 
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MI  D  STD-454 

Design  Discipline  Requirement  Hardware  Requirement 

Req.  54  -  Maintainability  Req.  7  -  Interchangeability 

Req.  36  -  Accessibility 

It  can  be  seen  that  the  subjects  of  “interchangeability”  and  “accessibility”  are  maintainability 
hardware  requirements. 

Having  determined  the  applicable  requirements  (still  using  Maintainability  as  an  example),  the 
functions  of  each  along  with  applicable  documents,  are  specified  in  the  general  standard  (MID 
STD-454)  as  follows: 

•  Maintainability 

Maintainability  Program  MID  STD-470 

Maintainability  Prediction  MIL-HDBK-472 

Maintainability  Verification  MIDSTD-47 1 

•  Accessibility 

Definition  of  Item  Levels,  Item  Exchange¬ 
ability  Models,  Related  Terms,  etc.  MID  STD-280 

This  hierarchy  of  documents  for  maintainability  is  depicted  in  Figure  2. 


4.2.2.2  Proposed  Application  for  Testability 

The  application  of  testability  to  a  family  of  specifications  necessitates  that  applicable 
requirements  be  passed  or  traced  from  specific  testability-oriented  documents  to  general 
ones.  As  with  maintainability,  the  starting  point  for  examining  this  flow  is  the  equipment 
level  specifications  (MIDE- 16400,  MIDT-28800,  etc.),  followed  by  a  general  requirements 
document  (MIDSTD-454)  that  specifies  hardware  and  program  (design  discipline)  requirements 
and  documents.  A  problem  becomes  apparent  in  the  application  of  testability  as  a  design 
discipline,  due  to  the  appearance  of  voids  in  testability  requirements  and  documentation.  For 
example,  the  equipment  specifications  (MIDE- 16400  and  MIDT-28800),  and  the  general 
standard  (MIDSTD-454)  do  not  presently  contain  testability  program  (design  discipline) 
requirements.  Therefore,  modification  of  MIDE- 16400  and  MIDT-28800  (et  al)  is  necessary  to 
draw  these  requirements  from  testability  documents.  Similarly  MIDSTD-454  will  need 
modification  to  include  testability  as  a  program  requirement  which  will  specify  a  family  of 
testability  program  functions  and  documents  as  follows: 


MIL- STD-454 


Design  Discipline  Requirement  Hardware  Requirement 

Requirement  XX  Testability  (Proposed)  Req.  32  -Test  Provisions 

Req.  YY  -  Interfaces  (Proposed) 

It  should  be  noted  that  Requirement  32  “Test  Provisions”  contained  in  MIL-STD-454  is  a 
testability  hardware  requirement  This  requirement  needs  to  be  updated  to  include  interface  require¬ 
ments  and/or  add  a  new  MIL-STD-454  Requirement  YY,  Interfaces.  This  allows  for  testability 
hardware  and  program  requirements  in  MID  STD-454  that  may  be  applied  in  parallel  as  with  maintain¬ 
ability  requirements.  The  W.  Keiner  study,  “A  Study  of  Testability  Standardization  for  Electronic 
Systems  and  Equipment”,  contains  the  testability  counterparts  to  maintainability.  The  data  contained 
in  the  proposed  W.  Keiner  document  could  be  used  to  develop  a  family  of  testability  documents  by 
function  similar  to  the  family  of  maintainability  documents.  This  family  of  MIL-STD-454  specified 
testability  program  documents  is  proposed  as  follows: 

•  Testability 

Testability  Program  Requirements 

Testability  Demonstration 

Testability  Analysis  and  Report 

•  Test  Provisions 

Test  Provisions  for  Electronic  Systems  and 
Associated  Equipment,  Design  Criteria  for  MID  STD-4 15 

•  Interfaces 

TBD  TBD 

The  proposed  hierarchy  of  testability  documents  is  illustrated  in  Figure  3. 


MID  STD- ABC  (Proposed) 
MIDSTD-DEF  (Proposed) 
MIDSTD-GHI  (Proposed) 


4.3  DESIGN  GUIDE  FUNCTION 

The  establishment  of  testability  as  a  design  discipline  will  require  more  than  specifying  requirements 
in  MIDSTDs,  Statements  of  Work(  SOWs)  and  performance  specifications.  The  design  engineers  that 
are  involved  in  the  performance  aspects  of  the  hardware  do  not  normally  have  background  or 
experience  in  testability.  The  design  engineers  that  have  testability  experience  are  usually  involved  in 
designing  test  equipment,  not  operational  hardware. 
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MIL- E-1 6400  MIL-T-28800 

EQUIPMENT  Electronic  Equipment.  Test  Equipment  for  use  with  MIL-X  (Others) 

SPECIFICATIONS  Naval  Ship  and  Shore  Electrical  and  Electronic 

General  Specification  Equipment,  General  Spec. 
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Figure  3.  Testability  Hierarchy  of  Documents  (Proposed) 


An  educational  process  will  naturally  occur  when  testability  is  implemented  as  a  design  discipline. 
This  educational  process  will  either  occur  through  trial  and  error  or  through  the  use  of  planned 
documentation.  The  use  of  testability  design  guides  will  help  bridge  this  potential  educational  gap. 


4.3.1  BUILT-IN- TEST  DESIGN  GUIDE  (BIT) 

The  Built-In-Test(BIT)  Design  Guide  is  currently  available  for  use  in  hardware  procurement  and 
may  be  updated  and  expanded  as  new  techniques  and  electronic  components  become  available. 
Although  the  BIT  Design  Guide  discusses  operational  monitoring,  it  is  primarily  associated  with 
Go/No-Go  testing.  The  classical  Go/No-Go  testing  concept  is  normally  based  on  reporting  that  the 
hardware  is  either  operational  or  non-operational. 

Early  BIT  efforts  were  developed  to  meet  the  maintenance  concept;  that  is,  how  to  determine  if  the 
equipment  is  ready  to  operate  and  how  to  isolate  a  failure.  How  well  it  was  working  was  not  a  driving 
consideration. 


4.3.2  OPERATIONAL  MONITORING  (ORMS)  DESIGN  GUIDE  (PROPOSED) 

Operational  monitoring  (part  of  ORMS  concept)  is  involved  with  how  well  the  equipment  is 
operating.  This  is  required  to  assess  the  degree  of  performance  and/or  degraded  performance  capability 
of  shipboard  hardware.  The  current  BIT  Design  Guide  does  not  stress  methods  of  implementing 
operational  monitoring.  The  addition  of  special  “Operational  Monitoring”  sections  or  the  develop¬ 
ment  of  a  separate  “Operational  Monitoring  Design  Guide”  will  help  in  the  testability  educational 
process. 


4.3.3  DESIGN  FOR  TESTABILITY  GUIDE 

The  conversion  of  the  W.  Keiner  “Design  for  Testability  Course”  to  a  written  design  guide  would 
be  a  beneficial  primary  document  for  the  testability  educational  process.  The  BIT  and  the  proposed 
ORMS  design  guides  would  be  related  specialized  documents. 


4.3.4  MIDSTD-1390,  LEVEL  OF  REPAIR 

MIL-STD1390,  Level  of  Repair  (LOR),  is  placed  in  the  same  category  as  the  design  guides 
because  of  its  value  in  performing  testability  design  trade-off  studies  by  the  hardware  development 
contractors.  Normally  the  LOR  is  used  during  the  acquisition  phase  in  determining  the  maintenance 
concept  During  the  design  of  the  hardware,  after  the  maintenance  concept  has  been  established, 
the  LOR  may  be  used  to  perform  those  hardware  design/cost  trade-off  studies  that  affect 
testability. 
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4.3.5  MIDSTD-1364,  STANDARD  GENERAL  PURPOSE  ELECTRONIC  TEST 
EQUIPMENT 

This  MID  STD  for  general  purpose  electronic  test  equipment  (GPETE)  was  placed  under  the 
design  guide  function  because  the  selection  of  GPETE  can  be  recommended,  but  not  dictated  by 
contract  The  GPETE  listed  in  MIDSTD-1364  would  be  a  “first-advice”  shopping  list  for  use  by 
the  hardware  development  contractor.  This  document  and  the  preceding  cited  and  proposed  guide- 
type  documents  would  be  discussed  in  the  statement  of  work  and  supplied  as  part  of  the  Request 
for  Proposal  (RFP)  package.  They  would  not  be  invoked  contractually,  but  would  be  supplied  as 
“information  only”  documents  to  be  used  by  the  contractor  as  he  sees  fit 

4.4  DOCUMENTATION 

The  following  documents,  as  shown  in  Figure  1,  are  grouped  under  the  Documentation 
function: 


•  MIL- STD-1519,  Test  Requirements  Document  preparation  of; 

•  MIL-STD-1345,  Data,  Measurement  in  Support  of  Maintenance,  Calibration,  and 
Repair  of  Electronic  Equipment 

•  MIL-STD-2076,  UUT  Compatibility  with  ATE,  General  Requirements  for, 

•  MID  STD- 207  7,  Test  Program  Sets,  General  Requirements  for, 

•  D-790,  Test  Program  Sets,  General  Requirements  for,  NESEC  SD  STD; 

•  MIDT-24309,  Technical  Support  Plan  for  Electronic  Equipment 

•  MIDSTD  21 1 1,  Technical  Repair  Standards  (4G  Repairables),  Preparation  of  (Draft); 

•  MIDT-24255,  Technical  Repair  Standards,  Submarines; 

•  MIDSTD-490,  Specification  Practices; 

•  MIDSTD-GHI  (Proposed),  Testability  Analysis  and  Report,  Preparation  of, 

•  MIDT-24424,  Technical/Maintenance  Overhaul  and  Repair  Standards; 

•  MIDSTD-31 10,  Restoration,  Overhaul,  and  Repair  of  Electronic  Equipment 

•  MIDSTD-1521,  Technical  Reviews  and  Audits  for  Systems,  Equipments,  and 
Computer  Programs. 

4.4.1  DOCUMENTATION  FUNCTION 

The  testability  documentation  function  relates  to  those  documents  that  require  the  drafting  or 
submittal  of  testability  related  data.  This  data  may  be  required  at  Preliminary  Design  Reviews 
(PDR),  Critical  Design  Reviews  (CDR),  and  audits;  in  addition  to  being  cited  by  Contract  Data 
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Requirements  List  (CDRL)  item  required  by  contract  The  types  of  testability  data  are  as  follows: 

•  Performance  Specification  (MID STD-490); 

•  Testability  Analysis  and  Report  (Proposed  MIDSTD-GHI); 

•  Test  Requirements  Data  (TRD); 

•  Testability  Demonstration  (Proposed  MIL-STD-DEF); 

•  Technical  Support  Plan  (TSP); 

•  Test  Program  Sets  (TPS); 

•  Technical  Repair  Standards  (TRS); 

•  Restoration,  Overhaul,  and  Repair  ( ROR); 

•  Technical  Reviews  and  Audits. 

The  flow  of  testability  related  data  will  generally  follow  the  top-to-bottom  sequence  depicted  in 
Figure  4,  Testability  Documentation  Tree.  A  discussion  of  these  testability  data  items  is  presented 
in  the  following  sections.  A  more  detailed  analysis  of  selected  documents  pertaining  to 
documentation  is  contained  in  Appendix  A,  Document  Survey. 

4.4. 1.1  Performance  Specification 

The  performance  specifications  are  formatted  per  MIL-STD-490.  MID  STD-490  is  placed 
under  the  documentation  section  (para  4.4)  for  this  report  because  these  specifications,  depending 
on  production  volume,  eventually  become  hardware  equipment  specifications.  MID  STD-490  was 
also  referenced  under  the  Technical  Requirements  section  (para.  4.2)  because  the  original  RFP 
performance  specifications  contain  testability  design  requirements.  MIDSTD-490  does  not 
currently  contain  specific  testability  sections  as  it  does  for  reliability  and  maintainability.  This  is 
required  to  establish  testability  as  a  design  discipline.  See  Section  4.2,  Hardware  Performance 
Specifications. 

4.4. 1.2  Testability  Analysis  and  Report 

The  Testability  Analysis  and  Report  (MIDSTD-GHI,  Proposed)  is  a  proposed  fall-out  from 
the  W.  Keiner  report,  “A  Study  of  Testability  Standardization  for  Electronic  Systems  and 
Equipment”.  This  analysis  will  provide  qualitative  and  quantitative  data  that  determine  the 
equipment’s  ability  (or  lack  of  ability)  to  pass  a  testability  demonstration  test  This  MID  STD  will 
contain  provisions  for  a  Testability  Analysis  Report  (TAR)  for  documenting  results  of  the 
testability  analysis.  The  TAR  will  be  presented  at  PDR/CDR,  audits,  and  will  be  delivered  per 
CDRL  item  instructions.  This  testability  analysis  is  in  addition  to  the  maintainability  analysis. 

The  prime  concern  will  be  the  determination  that  the  quantitative  changes  (deltas)  are  detectable 
at  the  interfaces  following  a  failure.  This  may  be  implemented  as  a  modified  Failure  Modes  and 
Effect  Analysis  (FMEA)  currently  in  use  by  reliability  engineers.  The  modified  portion  of  the 


-  18- 


FMEA  would  be  the  quantitative  changes  at  the  interface  in  addition  to  the  effect  of  the  failure. 
These  quantitative  changes  (deltas)  would  then  be  used  to  determine  the  tolerances  and  accuracy 
requirements  of  associated  sensing  equipment  This  is  the  lead-in  to  Test  Requirements  Data 
(TRD)  and  the  Testability  Demonstration  Plan  (proposed). 

4.4. 1.3  Test  Requirements  Data  (TRD) 

The  TRD  will  utilize  data  from  the  TAR  The  TRD  is  currently  covered  in  the  following  MIL 
documents: 

•  MIDSTD-1519,  Test  Requirements  Document  Preparation  of; 

•  MIDSTD-1345,  Data,  Measurement  in  Support  of  Maintenance,  Calibration,  and 
Repair  of  Electronic  Equipment 

•  MIDSTD-2076.  Unit  Under  Test  Compatibility  with  ATE,  General  Requirements  for. 

A  discussion  of  these  MIL- STDs  follows: 

MIDSTD-1519 

MILSTD-15 19  performs  the  same  general  function  as  Appendices  A,  C  and  D  of  MID  STD- 
2076.  MILSTD-15 19  is  not  set  up  to  use  ATLAS,  whereas  MID  STD- 2076  does. 

MIDSTD-1345 

MIDSTD-1345  works  with  MIDSTD-21 1 1,  “Technical  Repair  Standards  (4G  Repairables), 
Preparation  of’.  MIDSTD-1345  is  primarily  a  data  document  for  the  preparation  and  submission 
of  data.  This  document  is  primarily  associated  with  GPETE,  not  ATE. 

Requirements  for  test  procedures  are  specified  as  (a)  performance  test  procedures,  (b)  fault 
location  procedures,  and  (c)  alignment  procedures. 

a.  Performance  test  procedures  are  typified  by  those  procedures  developed  by  an  equipment 
contractor  for  qualification  and  acceptance  testing  of  the  operational  hardware.  Normally 
this  testing  is  not  performed  by  ATE. 

b.  Fault  location  procedures  are  typified  by  those  procedures  used  in,  or  in  conjunction  with, 
a  Technical  Repair  Manual.  Reference  is  not  made  relative  to  ATE  which  requires  TPS 
(Example:  MIDSTD-2077). 

c.  Alignment  procedures  are  typified  by  those  procedures  used  in  a  depot  operation 
(Technical  Repair  Manual/Calibration  Procedure). 

In  general,  MIDSTD-1345  is  not  specifically  set  up  for  ATE.  Data  is  specified,  but  not 
specifically  for  the  generation  of  TPS. 
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MIL- STD- 2076 


MIDSTD-2076  was  set  up  to  work  with  MIDSTD-2077,  Test  Program  Sets,  and  AR-10, 
(MIL-STD-2084,  Proposed),  "Maintainability  of  Avionics  Equipment  and  Systems,  General 
Requirements  for”.  This  MIUSTD  used  the  terms  “Avionics"  and  “WRA”  which  are  not 
suitable  for  shipboard  application.  These  avionic  terms  would  need  to  be  standardized  to  more 
general  terms  to  make  MIUSTD-2076  a  more  universal  document 

Section  5.1.1,  Design  for  Test  on  ATE.  The  contents  of  this  section  are  not  requirements,  but 
rather  recommendations  and  examples,  and  are  more  applicable  in  the  “Testability  Design 
Guide”  (Proposed). 

Section  5.1.2,  Test  Points,  apply  to,  and  should  be  inserted  in  MIDSTD-415,  “Test 
Provisions  for  Electronic  Systems  and  Associated  Equipment  Design  Criteria  for”. 

Section  6.1.2,  UUT  Documentation  Analysis  and  Hardware  Inspection,  are  more  functional  in 
the  following  proposed  documents: 

•  MIDSTD-GHI,  Testability  Analysis  and  Report 

•  MIUSTD-DEF,  Testability  Demonstration 

NOTE:  See  Management  and  Technical  Requirements  sections  of  this  study. 

Appendix  A,  Test  Requirements  Data,  with  companion  Appendices  C  and  D,  perform  the 
same  general  function  as  MIL-STD-15 19,  “Test  Requirements  Documentation,  Preparation  of’. 

Appendix  B,  Compatibility  Task  Scoring  Data:  A  scoring  or  rating  system  is  more  functional 
in  the  proposed  MIDSTD-GHI,  “Testability  Analysis  and  Report”.  This  type  of  analysis,  if 
performed  early  in  the  R&D  cycle  (prior  to  PDR),  can  influence  the  design.  If  performed  at  or 
after  the  qualification  testing,  its  function  becomes  passive. 

Appendices  C  and  D  are  as  described  in  Appendix  A. 

Appendix  E,  “Application  of  UUT  Failure  Rate  Data”:  This  type  of  data  can  influence  the 
design  of  the  hardware  and  should  be  available  prior  to  the  PDR  This  type  of  data  is  also  more 
functional  in  the  proposed  MIL-STD-GHI,  “Testability  Analysis  and  Report”. 

Appendix  F,  “Fault  Isolation  (FI)  Test  Requirement,  Fault  Isolation  Data”:  This  appendix 
performs  the  same  general  function  as  MIL- STD- 1 345.  The  major  differences  are  the  use  of 
functional  and  data  flow  charts  (ATLAS  Test  Procedures)  in  MIL-STD-2076,  while  MIDSTD- 
1345  calls  out  detailed  test  procedures. 

The  TRD  will  be  the  data  base  for  the  following  documents: 

•  Testability  Demonstration  Procedure 

•  Technical  Support  Plan  (TSP) 
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•  Test  Program  Sets  (TPS) 

4.4. 1.4  Testability  Demonstration  (MIL-STD-DEF)  Proposed 

The  content  for  Testability  Demonstration  Plan/Procedure/Report  documents  are  contained  in 
the  W.  Keiner  report  “A  Study  of  Testability  Standardization  for  Electronic  Systems  and 
Equipment”.  The  testability  demonstration  plan,  procedure  and  report  documents  will  be  CDRL 
items  that  will  document  the  testability  demonstration  test 

Testability  Demonstration  Plan 

The  Testability  Demonstration  Plan  delineates  the  test  requirements  and  the  general 
implementation  of  the  testability  testing  This  allows  for  a  meeting  of  the  minds,  between  the 
procuring  agency  and  the  contractor,  concerning  the  interpretation  of  the  test  requirements  and 
their  implementation.  The  Testability  Demonstration  Plan  is  expected  to  be  drafted  after  the  TAR 
and  prior  to  the  TRD  (see  Figure  4).  The  plan  will  be  submitted  for  approval  prior  to  the 
testability  demonstration  procedure. 

Testability  Demonstration  Procedure 

The  Testability  Demonstration  Procedure  will  follow  the  TRD  (see  Figure  4).  The  procedure 
will  specify  the  method  of  verifying  the  test  requirements  data. 

Testability  Demonstration  Report 

The  Testability  Demonstration  Report  follows  the  Testability  Demonstration  Test 


4.4. 1.5  Technical  Support  Plan  (TSP) 

The  TSP,  MIL-T-24309,  is  an  accumulation  of  all  data  that  are  pertinent  to  the  support  of 
equipment  Included  in  addition  to  testability  data  are: 

«  Maintenance  concept; 

•  Plan  for  maintenance; 

•  Reliability  design  data; 

•  Maintainability  design  data. 

The  testability  related  data  are  specified  as  ( 1 )  modular  construction  and  assembly  design  data,  and 
(2)  test  point  and  test  equipment  data.  The  types  of  data  specified  in  MIL-T-24309  are  normally 
available  as  CDRL  items  separately,  but  not  in  one  place.  In  the  event  MIDT-24309  is  to  be 
specified,  the  following  recommendation  is  submitted  in  the  testability  area: 

•  Modify  paragraph  3. 2. 1.5,  “Modular  Construction  and  Assembly  Design  Data”  and 

3. 2. 1.6,  “Test  and  Test  Facilities  Data”,  to  reference  information  contained  in  the  Test 
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Requirements  Data  (TRD)  as  specified  in  MIDSTD-1519,  MIL- STD- 1345,  or  MID 
STD-2076. 


4.4. 1.6  Test  Program  Sets  (TPS) 

The  TPS  documents  available  to  the  reviewer  were  MID  STD- 207 7  and  D-790.  MIDSTD- 
2077  is  a  NAVAIR  release  while  D-790  is  an  internal  document  prepared  by  NESEC.  Generally 
speaking,  the  D-790  standard  closely  parallels  MIDSTD-2077.  Major  differences  are: 

•  D-790  terminology  is  more  universal.  For  example,  it  does  not  use  restrictive  terms  such 
as  “avionics”  and  “NAVAIR”.  It  tends  to  avoid  use  of  MIDdocuments  in  specifying 
requirements  and  is,  therefore,  ambiguous; 

•  MIDSTD-2077  is  more  definitive  than  D-790.  It  makes  provisions  for  validation  of  the 
TPS  by  the  Government  agency,  whereas  D-790  is  weak  in  this  area. 

Neither  document  is  directly  applicable  to  shipboard  equipment  MIDSTD-2077  would  have  to 
be  made  less  restrictive  by  broadening  terminology  beyond  the  scope  of  typical  NAVAIR 
terminology,  and  D-790  would  have  to  be  more  definitive  in  the  area  of  requirements  to  be 
considered  for  status  as  a  released  MIDSTD. 

NOTES:  1.  MIDSTD-2077  is  a  companion  document  to  MIDSTD-2076  as  indicated  by 
paragraph  5.10.1.1,  “Program  Design  Data  (PDD)”  of  MIDSTD-2077. 

2.  It  is  important  to  assure  the  software  (TPS)  is  available  in  time  to  perform  the 
testability  demonstration.  This  potential  time  phasing  problem  will  be  made  an 
item  to  be  included  in  the  testability  demonstration  plan.  An  ideal  situation  is 
to  have  the  hardware  design  contractor  develop  the  TPS  to  match  the  Govern¬ 
ment  agency’s  test  requirements. 

Detailed  comparisons  of  MIDSTD-2077  and  D-790  are  contained  in  Appendix  A,  Document 
Review. 


4.4. 1.7  Technical  Repair  Standards  (TRS) 

The  TRSs  are  covered  by  the  following  documents: 

•  MIDSTD-21 11,  Technical  Repair  Standards  (4G  Repairables),  Preparation  oft 

•  MIDT-24255,  Technical  Repair  Standards  (For  Submarines  Only); 

•  MIDT-24424,  Technical/Maintenance  Overhaul  and  Repair  Standards. 

The  discussion  of  these  documents  follows. 

MIDSTD-21 11 

MIDSTD-21 1 1  covers  repair  at  the  depot  level  of  maintenance.  This  document  was  set  up  to 
work  with  MIDSTD-1345  and  MIDSTD-1364.  MIDSTD-21 1 1  does  not  make  specific 
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provisions  for  ATE  and  the  supporting  TPSs.  MIL- STD- 1345  is  a  TRD  document,  but  not  for 
ATE.  The  addition  of  an  ATE  section  which  references  MILSTD-2076  (TRD)  and  MILSTD- 
2077  (TPS)  would  augment  MILSTD-21 1 1 .  A  more  efficient  approach  would  be  to  cancel  MIL 
STD- 1345  and  use  MILSTD-2076  for  the  TRD  data,  for  the  use  of  ATE  and  GPETE.  This  is 
based  on  the  following: 

•  Appendix  E  of  MILSTD-2076  covers  the  non- ATE  testing  in  terms  of  ATLAS  in  lieu 
of  detailed  test  procedures; 

•  The  detailed  test  procedures  of  MILSTD-I345  call  for  specific  test  equipment  which  has 
the  following  limitations: 

—  Identification  of  test  equipment  by  model  number  (vendor)  is  limited  and  shows 
preference  by  manufacture. 

—  A  problem  of  substitution  results  in  when  the  specific  test  equipment  is  not  available 
at  the  depot  in  question. 

-  The  supporting  TRD  (ATLAS)  will  not  be  available  in  event  ATE  is  to  be  used 
later. 

MILT-24255 

MILT-24255  was  drafted  for  the  overhaul  of  submarines,  and  does  not  apply  directly  to 
repair  of  hardware  at  a  depot  facility. 

MILT-24424 

MILT-24424  was  drafted  for  the  overhaul  of  ships.  This  document  is  similar  to  MILT- 
24255. 


4.4. 1.8  Restoration,  Overhaul,  and  Repair  (ROR) 

MILSTD-31 10,  Restoration,  Overhaul,  and  Repair  of  Electronic  Equipment,  is  listed  under 
documentation  because  testing  is  a  part  of  it  In  the  tier  of  documents,  MILSTD-31 10  uses  TRSs 
which  are  covered  in  MILSTD-21 1 1,  MILT-24255,  and  MILT-24424. 


4.4. 1.9  General  Purpose  Electronic  Test  Equipment  (GPETE) 

The  general  purpose  electronic  test  equipment  (GPETE)  is  shown  in  Figure  4,  Testability 
Documentation  Tree,  for  those  cases  where  ATE  is  not  used.  The  GPETE  data  is  specified  in  the 
TRD  and  used  in  the  TRS  and  ROR  documents.  The  TRS/ROR  documents  utilize  the  TRD, 
GPETE  and  TPS  data. 
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Figure  4.  Testability  Documentation  Tree 
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4.4.1.10  Design  Reviews  and  Audits 

MIL-STD-1521  checklists  contain  maintainability,  reliability  and  other  data  to  be  covered 
during  design  reviews  and  audits.  No  such  provisions  exist  for  testability. 


4.5  VALIDATION 

The  following  documents  are  grouped  under  validation,  as  shown  in  the  Testability  Matrix, 
Figure  1: 

•  MIDSTD-471,  Maintainability  Verification/Demonstration/Evaluation 

•  MIL-STD-DEF  (Proposed),  Testability  Demonstration 

Testability  validation  is  performed  as  part  of  hardware  qualification  and  acceptance  testing. 
This  also  includes  testability  software.  The  purpose  of  these  tests  is  to  prove  that  the  hardware 
design  meets  the  testability  requirements.  Classically,  the  testability  validation  has  been 
approached  through  the  maintainability  demonstration  test  per  MIL-STD-471.  The  maintain¬ 
ability  demonstration  ruling  parameter  is  the  mean- time- to- repair  (MTTR).  The  MTTR  parameter 
can  be  reduced  to  the  following  time  steps: 

Function  Using 

Detect  Test  equipment  or  BIT 

Isolate  Test  equipment  or  BIT 

Remove/Replace  Technician 

Verify  (retest)  Test  equipment  or  BIT 

The  faster  the  test  equipment  or  BIT  operates,  the  more  time  is  allowed  to  remove/replace.  The 
total  MTTR  time  is  the  sum  of  the  individual  function  times.  MIDSTD-471  restricts  the  time  in 
which  the  test  equipment  must  perform.  This  is  to  the  direct  detriment  of  the  important  testability 
factor  of  how  well  it  performs. 

Also,  the  selection  of  maintenance  tasks  for  the  maintainability  demonstration  is  based  on  a 
stratification  procedure;  the  intent  being  to  ‘‘(a)  allow  for  the  selection  of  maintenance  tasks  in 
such  a  manner  that  the  selection  simulates  the  failure  frequency  of  the  test  unit  in  actual  operation 
(units  with  low  MTBFs  will  be  selected  more  frequently  than  units  with  higher  MTBFs),  and  (b) 
ensure  that  a  proportionately  representative  sample  of  task  types/ times  are  selected."  A  typical 
test  is  based  on  50  tasks  which  are  selected  from  200  tasks.  The  200  tasks  along  with  the  method 
of  implementation  are  normally  selected  and  submitted  by  the  contractor.  The  customer  will  select 
50  tasks  from  the  200.  Since  the  contractor  selects  the  200  tasks,  there  is  little  chance  that  his 
hardware  will  fail  the  maintainability  demonstration.  With  the  contractor  controlling  the  outcome 
of  the  maintainability  demonstration  and  the  controlling  factor  being  the  MTTR  parameter,  a 
question  exists  as  to  how  testable  the  hardware  design  really  is.  The  information  contained  in  the 
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W.  Keiner  study,  when  formatted  into  a  testability  demonstration  MIl^STD,  would  overcome  the 
shortcomings  of  MIDSTD-471.  This  is  not  a  recommendation  to  scrap  MIL-STD-471,  but  to 
augment  the  maintainability  demonstration  with  a  testability  demonstration  either  concurrently  or 
at  a  different  time. 


4,6  INTERFACES 

The  following  documents  are  grouped  under  the  interface  function  in  the  Testability  Matrix, 
Figure  1: 

•  MIDSTD-XXXX  (NOSC  Proposed),  System  Level  Performance  Monitoring,  General 
Requirements  for, 

•  MIL-STD-1657,  Switch  Equipment,  Combat  System,  Command  and  Control,  and  Fire 
Control,  Requirements  for, 

•  Ships  Data  Multiplexer  System  (SDMS)  (I/O  Modules),  Equipment  Specification; 

•  MIDSTD-1553,  Aircraft  Internal  Command/Response  Time  Division  Multiplex  Data 
Bus; 

•  MIL-STD-1397,  Input/Output  Interfaces,  Standard  Digital  Data,  Navy  Systems; 

•  DOD-STD-1399,  Interface  Standard  for  Shipboard  Systems; 

•  IEEE  STD  488,  1978  Standard  Digital  Interface  for  Programmable  Instrumentation; 

•  On-Line/Off-Line  Interface  (Proposed); 

•  MIL-STD-1326,  Test  Points,  Test  Point  Selection  and  Interface  Requirements  for 
Equipment  Monitored  by  Shipboard  On-Line  Automatic  Test  Equipment 

Although  this  list  of  interface  documents  is  not  complete,  it  is  an  indication  of  the  various 
documents  that  have  been  developed  in  an  attempt  to  standardize  interfaces.  The  hardware  inter¬ 
face  can  be  categorized  as  physical,  performance,  and  contractual.  The  physical  interfaces  are 
determined  by  dimension,  connector  callout,  pin  designation,  fluid  coupling,  etc.  The  performance 
interfaces  comprise  electrical,  mechanical  and  information  format  and  content  Contractual 
interfaces  are  physical  and  performance  interfaces  that  can  be  controlled  and  checked  by 
qualification  and  acceptance  testing  The  hardware  interfaces  (physical  and  performance)  are 
contractual  During  acceptance  testing  the  information  available  for  determining  compliance  is 
normally  at  the  surface  of  the  package  and  through  connectors  and/or  couplings.  Because  of  this 
contractor-customer  contractual  interface,  the  standardization  of  interfaces  for  specific  types  of 
hardware  will  result  in  reduced  life  cycle  costs. 


-26- 


The  selection  of  an  interface  for  use  between  equipment  and  a  data  bus  (on-line  and  off-line)  is 
based  on  the  following: 

•  Performance  - 

—  Data  rates  determine  the  type  of  bus.  Parallel  buses  are  faster  while  serial  buses  are 
slower  but  better  over  long  distances. 

—  Information  content  across  the  interface  will  determine  the  word  length  requirements. 
--  Electrical  interfaces  are  controlled  by  the  bus  I/O  modules. 

•  Physical  - 

-  Physical  interfaces  are  controlled  by  the  bus  I/O  modules. 

•  Contractual 

—  Contractual  interfaces  are  those  (performance  or  physical)  that  can  be  controlled  and 
checked  by  qualification  and  acceptance  testing. 

Because  of  the  varied  performance  requirements,  no  one  interface  will  meet  the  requirements 
of  all  users.  A  standard  shopping  list  of  interfaces  should  be  available  for  selection  to  meet  the 
user’s  performance  requirements.  DOD-STD-1399,  being  a  general  standard,  and  MIL-STD- 
XXXX  (NOSC  Proposed)  are  the  logical  places  to  place  these  shopping  lists.  This  shopping  list 
would  reference  the  other  data  bus  interface  standards  by  type  and  use. 

MIL-STD-XXXX  (NOSC) 

MIL-STD-XXXX  (NOSC  Proposed)  presents  a  methodology  of  developing  a  set  of  sections 
or  MIL-STDS  that  are  devoted  to  the  standardization  of  performance  monitoring  for  classes  of 
hardware.  Three  levels  of  interface  are  proposed  to  allow  flexibility  in  meeting  the  user’s  tolerance 
requirements. 

MIL- STD-1657 

MIL-STD-1657  is  primarily  a  physical  interface  document  that  is  referenced  by  MIDS- 
1 7000.  This  is  applicable  to  combat  systems. 

Ships  Data  Multiplexer  System  (SDMS)  Equipment  Specification 

The  SDMS  I/O  modules  standardize  the  interface  between  the  SDMS  and  shipboard 
equipment  This  will  be  applicable  to  ships  that  will  have  the  SDMS  system. 

MILr  STD-1553 

MIL-STD-1553  specifies  a  type  of  series  data  bus.  This  was  developed  primarily  to  reduce  the 
amount  of  hardwiring  in  aircraft 
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MIL- STD- 1397 


MIUSTD- 1 397  is  an  interface  document  primarily  concerned  with  computers  and  the 
following  digital  bus  interfaces: 

•  Type  A  (NTDS  slow); 

•  Type  B  (NTDS  fast); 

•  Type  C  (ANEW); 

•  Serial. 

DOD- STD- 1399 

DOD-STD-1399  with  supporting  sections  specifies  interfaces  for  shipboard  equipment  These 
interfaces  include  the  following  types: 

•  Support  Services  (Environmental): 

•  Electronic  Information  (Functional); 

•  Controlled  Factors  (Environmental); 

•  Weapons  Control  (Functional); 

•  Uncontrolled  Environment  (Environmental); 

•  Operational  Factors  (Environmental); 

•  Electrical  Information  (Functional); 

•  Mechanical  Information  (Functional). 

The  physical  types  of  interfaces  are  specified  under  “controlled  factors  (environmental)”.  An 
example  of  this  is  Digital  Computer  Grounding  ( Section  406,  MIL-STD-1399)  where  MIL-C-915 
cable  is  specified  This  is  similar  to  the  approach  used  by  MIL-STD-1657  which  specifies 
physical  parameters,  but  is  a  subtier  document  of  MIUS- 17000.  MIL-S- 17000  is  an  equipment 
specification  whereas  Section  406  is  a  subset  of  a  MI U Standard  (MIUSTD-1399). 

IEEE  STD-488 

IEEE  STD-488  is  a  parallel  digital  interface  bus  that  was  developed  for  a  family  of 
programmable  test  equipment 

On-Line/OfF-Line  Interface  (Proposed) 

The  establishment  of  a  standard  on-line/off-line  interface  would  apply  to  new  designs.  This 
would  require  a  means  to  balance  performance  monitoring  with  classical  BIT.  The  differences 
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between  performance  monitoring  and  the  classical  BIT  are  well  explained  in  Section  3.2.7 
(Tolerances)  of  the  Built-In-Test  Design  Guide,  June  1980.  Classical  BIT  is  more  concerned  with 
catastrophic  failure  and  isolation  to  support  a  maintenance  concept  than  how  well  the  equipment 
is  working.  Performance  monitoring  (how  well  the  equipment  is  working)  requires  tighter 
tolerances  than  the  classical  BIT. 

A  problem  exists  in  preserving  data  integrity  where  tight  tolerances  are  required.  The  sensed 
information  first  exists  in  electrical  (volts,  amps,  frequency,  etc.)  or  physical  (pressure, 
temperature,  etc.)  form.  This  information  becomes  data  only  when  it  is  interpreted  by  a  person  or 
machine.  This  occurs  in  the  following  ways: 

•  A  person  reads  an  instrument  display  and  interprets  the  data  (the  tolerances  of  the  data 
are  based  on  the  sensor  and  display  meter); 

•  The  information  is  sensed  and  sent  over  a  transmission  line  prior  to  being  interpreted  as 
above.  This  sending  of  information  over  transmission  lines  causes  degradation; 

•  The  information  is  sensed  and  converted  to  a  digital  form  (data)  prior  to  sending  over  the 
transmission  line.  This  tends  to  preserve  the  data  and  ease  the  problem  when  the  data 
must  be  sent  to  a  place  removed  from  the  hardware. 

In  the  future,  the  interpretation  of  sensed  performance  data  can  be  used  to  detect  and/or 
predict  failures.  The  prediction  of  failures  permits  implementation  of  preventive  maintenance  prior 
to  catastrophic  failures.  The  classical  BIT  would  be  used  to  isolate  failures  at  the  operational 
level.  This  approach  will  probably  require  a  standard  family  of  sensors  that  converts  information 
directly  to  the  digital  data  state.  The  proposed  on-line/off-line  interface  MIL- STD  would  contain 
this  family  of  standard  sensors  along  with  methods  for  their  application  and  evaluation.  The  pri¬ 
mary  emphasis  would  be  on  the  characteristics  at  the  interface.  This  proposed  interface  MIL- STD 
or  section  would  be  a  subset  of  MIL-STD-XXXX  (NOSC  Proposed).  MIL-STD-XXXX 
subsets  would  also  cover  the  types  of  data  that  are  required,  by  equipment  class,  to  have  perfor¬ 
mance  monitoring.  The  projected  result  of  the  increase  in  LSI  technology,  including  self-test,  will 
allow  for  greater  self-contained  BIT  in  future  military  hardware.  This  increased  capacity  will  allow 
the  BIT  to  perform  much  of  what  an  off-line  tester  currently  does. 


MIL- STD- 1326 

MIDSTEM  326  should  be  replaced  by  MIDSTDXXXX  (NOSC  Proposed).  The  sections 
on  selection  of  test  points  and  characteristics  fit  better  in  MIL-STD-415.  The  deliverable  items 
are  Test  Requirements  Data  (TRD),  covered  in  MID  STD-2076.  Since  MIDSTD-1326  was 
drafted  in  the  late  1960s,  the  interface  portion  needs  updating  and  will  fit  better  in  MIDSTD- 
XXXX  (NOSC  Proposed). 
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5.0  CONCLUSIONS 


5.1  GENERAL 

Established  design  disciplines  such  as  performance,  maintainability,  etc.,  are  adequately 
backed  by  military  specifications  and  standards.  The  subject  of  “testability”  as  a  design  discipline 
does  not  currently  exist  in  military  specifications  and  standards  for  use  in  procurement  Many 
military  documents  exist  old  and  recent  (MID  STD-41 5,  MIL-STD-1326,  MIL-STD-2076,  etc.), 
that  contain  testability  hardware  requirements,  but  not  testability  as  a  design  discipline.  Also, 
conflict  and  overlap  exist  between  different  documents  when  different  agencies  have  developed 
their  own  document  to  perform  an  identical  function.  The  specific  conclusions  (paragraph  5.2)  are 
presented  by  the  following  testability  functions: 

•  Management 

•  Technical  Requirements 

•  Design  Guide 

•  Documentation 

•  Validation 

•  Interfaces 

5.2  SPECIFIC  CONCLUSIONS 

Management  *  Testability  does  not  currently  exist  as  a  design  discipline  as  do  the  disciplines 
of  performance,  maintainability,  safety,  etc.  Heretofore,  testability  has  been  implemented  through 
the  maintainability  design  discipline  using  MIL- STD-4 70  as  the  management  document 
However,  the  satisfaction  of  contractual  maintainability  goals  (e.g.,  MTTR)  does  not  in  and  of 
itself  ensure  a  highly  testable  design.  A  similar  document  for  management  of  testability  by  an 
R&D  contractor  is  required  to  enhance  testability  interests.  These  management  elements  are 
available  in  the  W.  Keiner  document,  “A  Study  of  Testability  Standardization  for  Electronic 
Systems  and  Equipment”,  and  are  proposed  as  MID  STD- ABC,  “Testability  Program 
Requirements”. 

Technical  Requirements  -  Since  testability  does  not  exist  as  a  design  discipline  in  equipment 
specifications  and  general  standards,  MIL-STD-490,  “Specification  Practices”,  does  not  currently 
make  provisions  for  testability  as  a  specification  item  as  it  does  with  maintainability  and 
reliability.  Also,  general  standard,  MID  STD-454,  does  not  have  a  requirement  that  is  dedicated 
to  the  subject  of  testability.  Requirement  32,  Test  Provisions,  is  associated  with  the  hardware 
requirement;  however,  it  doesn’t  address  the  design  discipline  of  testability.  As  a  point  of  illustra¬ 
tion,  MIDSTD-454  specifies  the  maintainability  requirement  by  function  as  follows: 

•  MIDSTD-470  Maintainability  Program 

•  MIDHDBK-472  Maintainability  Prediction 
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•  MIL-STD-471  Maintainability  Verification 

As  previously  noted,  this  approach  to  Testability  does  not  currently  exist  in  released  form.  MID 
STD- XXX  Proposed,  W.  Keiner,  “Testability  Acquisition  Program  Requirements  for  Electronic 
Equipments  and  Systems”,  covers  these  topics  in  a  single  document 

MID  STD-41 5,  Test  Provisions  for  Electronic  Systems  and  Associated  Equipment  Design 
Criteria,  contains  a  mix  of  design,  management  and  documentation  requirements. 

Design  Guide  -  A  design  guide  is  not  a  contractual  document  during  the  procurement  of 
hardware.  The  design  guide  is  supplied  to  the  hardware  contractor  as  an  instrument  to  disseminate 
information. 

The  BIT  design  guide  fulfills  this  information  criteria  for  the  classical  case.  The  classical  BIT 
was  developed  to  meet  the  requirements  of  the  maintenance  concept  Due  to  this  fact  BIT  is 
limited  to  GO/NO-GO  testing  that  determines  equipment  readiness  for  operation  and  for  failure 
localization.  It  does  not  tell  how  well  the  equipment  is  working. 

A  design  guide  for  operational  monitoring  does  not  currently  exist  An  operational  monitoring 
design  guide  that  depicts  methods  of  implementing  performance  monitoring  would  support  the 
ORMS  concept 

The  W.  Keiner  document  Design  for  Testability  Course,  depicts  the  methodology  for 
implementing  testability  into  the  hardware.  This  course  would  fulfill  the  criteria  for  disseminating 
overall  testability  information  provided  that  contractor  personnel  would  have  access  to  it  An 
overall  testability  design  guide  will  aid  in  the  education  process. 

MIDSTD-1 390,  Level  of  Repair,  may  be  used  in  making  testability  design  trade-off  studies. 

MIDSTD-1 365,  Standard  General  Purpose  Electronic  Test  Equipment  provides  a  shopping 
list  for  use  by  hardware  contractors. 

Documentation  -  The  following  types  of  documentation  are  associated  with  testability: 

•  Performance  Specification  (MIL-STD-490) 

•  Testability  Analysis  and  Report  (Proposed) 

•  Test  Requirements  Data  (TRD) 

•  Testability  Demonstration  (Proposed) 

•  Technical  Support  Plan  (TSP) 

•  Test  Program  Sets  (TPS) 

•  Technical  Repair  Standards  (TRS) 

•  Restoration,  Overhaul,  and  Repair  (ROR) 


-31  - 


Although  MIL- STD-490  was  discussed  under  technical  requirements,  it  is  also  applicable  here 
because  the  performance  specifications  may  become  an  equipment  specification  depending  on  the 
production  volume. 

A  test  analysis  and  report  MID  STD  does  not  currently  exist  in  released  form.  This  type  of 
analysis,  if  performed  during  the  hardware  development,  would  influence  the  design. 

Overlap  and  conflict  exists  between  the  following  Test  Requirements  Data  (TRD)  MIL-STDs: 

•  MIL-STD-1519 

•  MIDSTD- 1345 

•  MID  STD-2076 

MID  STD-2076  is  the  more  current  of  the  three  documents.  This  MIDSTD  pertains  to  the  use  of 
ATE,  ATLAS  and  Test  Program  Sets  (TPS). 

The  callout  for  CDRL  items  and  DIDs  pertaining  to  testability  demonstration  plans, 
procedures  and  reports  does  not  currently  exist.  A  testability  demonstration  MIL-STD  similar  to 
the  maintainability  demonstration,  MIDSTD-471,  does  r.ot  exist  in  released  form. 

The  technical  support  plan  (TSP)  document,  MIDT-24309,  does  not  contain  reference  to  the 
Test  Requirements  Data  (TRD)  as  specified  in  MIDSTD-1519,  MIDSTD- 1345  or  MID  STD- 
2076. 

The  test  program  sets  (TPS)  document,  MID  STD- 2077,  being  slanted  towards  avionics,  is 
not  sufficiently  universal  for  use  outside  NAVAIR. 

The  following  technical  repair  standard  (TRS)  documents  were  reviewed  as  part  of  this  study: 

•  MIDSTD-21 11,  Technical  Repair  Standards  (4G  repairables),  Preparation  of 

•  MIDT-24255,  Technical  Repair  Standards  (For  Submarines  Only) 

•  MIDT-24424,  Technical/Maintenance  Overhaul  and  Repair  Standards 

These  documents  are  specialized  and  perform  different  functions.  MIDSTD-21 1 1  is  for  depot 
repair.  MIDT-24255  is  for  the  overhaul  of  submarines  and  MIDT-24424  is  for  the  overhaul  of 
ships.  MIDSTD-21 1 1  does  not  make  specific  provisions  for  ATE  and  TPS.  MIDSTD-31 10, 
Restoration,  Overhaul,  and  Repair  of  Electronic  Equipment,  is  listed  under  documentation 
because  testing  is  a  part  of  it  In  the  tier  of  documents,  MIL-STD-31 10  uses  TRS  which  are 
covered  in  MIDSTD-21 11,  MIDT-24255,  and  MIDT-24424. 

The  general  purpose  electronic  test  equipment  (GPETE)  is  shown  in  Figure  4,  Testability 
Related  Data  Flow,  for  those  cases  where  ATE  is  not  used.  The  GPETE  data  are  specified  in  the 
TRD  and  used  in  the  TRS  and  ROR  documents.  The  TRS/ROR  documents  utilize  the  TRD, 
GPETE  and  TPS  data. 
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Validation  -  MIL-STD-47 1  places  its  priority  on  how  long  test  equipment  must  operate.  This 
opposes  the  more  important  testability  aspects  of  how  effectively  the  test  equipment  works.  The 
controlling  factor  is  the  Mean  Time  To  Repair  (MTTR)  parameter.  A  document  does  not  exist  in 
released  form  to  control  the  demonstration  of  the  testability  of  the  hardware. 

Interfaces  -  Many  documents  have  been  developed  in  an  attempt  to  standardize  physical, 
performance  and  contractual  interfaces.  Because  of  the  varied  requirements  by  different  users,  a 
single  interface  will  not  satisfy  everyone.  A  limited  shopping  list  of  interfaces  is  the  logical 
solution.  DOD-STD-1399,  being  a  general  standard,  is  the  logical  place  to  put  the  shopping  list 

A  MIL-Document  that  deals  with  the  concept  of  performance  monitoring  does  not  currently 
exist  in  released  form.  The  existing  testability  related  MIDdocuments,  with  the  exception  of  MILr 
STD- 1326,  deal  with  satisfying  the  maintenance  concept  The  maintenance  concept  deals  with 
how  the  equipment  is  operating  as  well  as  with  isolating  failures.  It  does  not  address  how  well  it  is 
working.  MIL-STD-1326  does  not  meet  the  current  state  of  the  art  in  testability  technology. 

MIL-STD-XXXX  (NOSC  Proposed),  “System  Level  Performance  Monitoring,  General 
Requirements”,  addresses  the  subject  of  performance  monitoring.  A  methodology  for  developing  a 
set  of  sections  or  MID  STDs  by  classes  of  shipboard  hardware  for  performance  monitoring  is 
presented  in  MIDSTD-XXXX.  Performance  monitoring  requires  tighter  tolerances  than  the 
testing  associated  with  meeting  the  maintenance  concept  This  will  require  a  family  of  int  'face 
devices  that  transfers  the  physical  (pressure,  temperature,  etc.)  and  electrical  performance  ..olts, 
frequency,  etc.)  parameters  directly  to  the  information  state  (digital).  An  On- Line/ Off- Line 
interface  document  does  not  currently  exist  to  document  and  specify  this  family  of  interface 
devices.  The  projected  increase  in  LSI  technology,  including  self-test,  will  permit  greater  self- 
contained  BIT  in  future  hardware.  This  increased  capability  will  allow  the  BIT  to  perform  much 
of  what  an  off-line  tester  currently  does.  This  future  BIT  capability  will  be  part  of  the  family  of 
interface  devices. 


6.0  RECOMMENDATIONS 


6.1  GENERAL 

The  establishment  of  testability  as  a  design  discipline  will  require  generation  of  a  family  of 
testability  documents  that  can  be  applied  when  procuring  hardware.  This  will  require: 

•  Generation  of  new  military  documents; 

•  Cancellation  of  some  military  documents; 

•  Modification  of  some  military  documents. 

The  recommended  hierarchy  of  testability  documents  relative  to  the  procurement  of  hardware 
is  depicted  in  Figure  5.  The  following  footnotes  in  Figure  5  apply  as  follows: 

NOTES:  1 .  Modify  existing  document  -  Change  the  existing  document  to  avoid  conflict 
with  other  documents  and/or  to  specify  testability  requirements. 
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2.  Develop  New  Document  -  A  new  MIDdocument  is  required  to  specify 
testability  requirements  and/or  complete  the  family  of  testability  documents 

The  generation,  cancellation  and/or  modification  of  each  applicable  military  document  is 
discussed  in  paragraph  6.2  under  the  following  testability  functions: 

•  Management 

•  Technical  Requirements 

•  Design  Guide 

•  Documentation 

•  Validation 

•  Interfaces 


6.2  SPECIFIC  RECOMMENDATIONS 

Management  -  A  Testability  Program  document  (MIL-STD-ABC,  Proposed),  similar  to 
MIL- STD-470  for  maintainability,  should  be  developed.  This  testability  program  document  would 
be  derived  from  the  program  elements  contained  in  the  W.  Keiner  work,  “A  Study  of  Testability 
Standardization  for  Electronic  Systems  and  Equipment”. 

Technical  Requirements  •  The  following  will  be  required  to  establish  testability  as  a  design 
discipline  and  develop  a  family  of  MIL-documents  that  passes  testability  requirements  to  allied 
documents: 

•  Modify  MIDSTD-490  to  include  testability  requirements; 

•  Modify  MIDE- 16400,  MIDT-28800,  etc.,  to  include  testability  requirements,  and  to 
specifically  include  a  reference  to  MID  STD-45  4; 

•  Modify  MIDSTD-454  to  include  requirement  sections  for  testability  and  interfaces.  The 
testability  requirement  would  call  out  the  following  MID  STDs  proposed: 

-  Testability  Program  Requirements  (MIDSTD-ABC) 

-  Testability  Analysis  and  Report  (MIDSTD-GHI) 

-  Testability  Demonstration  (MIDSTD-DEF) 

NOTE:  The  three-document  approach  is  in  parallel  with  that  approach  used  by 

maintainability.  The  proposed  interface  section  for  MIDSTD-454  would  reference 
MIDSTD-1657  and  possibly  MIDSTD-1399. 


•  Modify  MIL- STD-4  IS  to  contain  only  design  and  hardware  requirements.  The  manage¬ 
ment  information  should  be  in  the  proposed  MID  STD- ABC  “Testability  Program 
Requirements".  The  documentation  requirements  should  be  in  the  TRD  document; 

•  Technical  requirements  as  well  as  the  test  point  placement  and  compatibility  aspects  of 
testability  must  be  stated  for  the  build-in  test  Whether  or  not  their  statement  is  best 
promulgated  under  a  single  document  (modified  MID  STD-4 1 5)  or  under  two  documents 
is  unresolved  at  this  time. 

Design  Guide  -  The  following  testability  design  guides  should  be  developed  to  aid  in  the 
testability  educational  process: 

•  Performance  Monitoring  Testability  Design  Guide  which  presents  methods  and  examples 
of  performance  monitoring  in  hardware; 

•  An  overall  Design  for  Testability  Guide  should  be  developed.  The  conversion  of  the 
W.  Keiner  document,  “Design  for  Testability  Course”,  to  a  written  test  will  fulfill  this 
need. 

Documentation  -  The  following  modifications  are  needed  in  the  testability  documentation 
area: 

•  Modify  MID  STD-490  to  include  testability  requirements; 

•  Develop  a  test  analysis  and  report  (MIDSTD-GHI)  utilizing  data  from  the  W.  Keiner 
report,  “A  Study  of  Testability  Standardization  of  Electronic  Systems  and  Equipment"; 

•  Develop  a  test  requirements  data  (TRD)  MID  STD  (TBD)  utilizing  MID  STD-2076  as  a 
base.  This  would  replace  MIDSTD-1519  and  MIDSTD-1345; 

•  Modify  MIDT-24309  to  reference  test  requirements  data  (TRD)  as  specified  in 
MIDSTD-1519,  MIDSTD-1345  or  MIDSTD-2076; 

•  Modify  MIDSTD-2077  to  cover  more  than  avionics; 

•  Modify  MIDSTD-21 1 1  to  cover  ATE  and  TPS; 

•  Develop  CDRL  Item  and  supporting  DIDs; 

•  Modify  MIDSTD-1521  to  include  testability  in  checklists  for  the  design  reviews  (SRR, 
SDR,  PDR,  CDR)  and  audits  (FCA,  PCA). 

Validation  -  Develop  a  testability  demonstration  MIDSTD-DEF  similar  to  the  maintain¬ 
ability  demonstration  document,  MIL-STD-471.  Utilize  data  from  the  W.  Keiner  report,  “A 
Study  of  Testability  Standardization  for  Electronic  Systems  and  Equipment”.  This  will  include 
the  testability  demonstration  plan,  procedure  and  report 

Interfaces  -  The  following  recommendations  are  submitted  to  standardize  interface 
requirements. 


-38- 


•  Utilize  DOD-STD-1399  as  the  general  standard  for  interfaces. 

•  Cancel  MIDSTD-1326.  The  section  on  test  points  and  characteristics  should  be  in 
MIL- STD-41 5.  The  deliverable  items  are  Test  Requirement  Data  (TRD)  and  should  be  in 
MIL-STD-2076.  The  remainder  is  obsolete. 

•  Develop  MIL-STD-XXXX  (NOSC  Proposed)  to  replace  MID  STD- 1326. 

•  Develop  an  On-Line/Off-Line  Interface  document  that  specifies  a  family  of  interface 
devices  that  convert  physical  and  electrical  performance  parameters  to  an  information 
state  (digital). 

Implementation  -  What  has  been  presented  herein  in  a  roadmap  for  the  development  of  the 
MIL-documents  required  to  facilitate  making  testability  happen  in  military  equipment  The  next 
logical  question  is,  “which  roads  do  we  take  and  when?”  The  functional  partitioning  utilized  in 
this  study  is  also  useful  for  dividing  the  work  that  should  follow.  Each  functional  area  can  be 
approached  separately  (with  due  attention  to  interactions)  for  the  definition  of  work  to  be  done 
and  for  the  generation  of  documents. 

The  following  program  is  recommended.  Each  functional  area  should  be  investigated  more 
deeply  to  turn  this  generalized  plan  into  a  more  detailed  implementation  plan.  These  plans  should 
be  sufficiently  detailed  to  serve  as  a  basis  to  effectively  estimate  the  time  and  effort  required  to 
achieve  the  final  goal.  Specifically,  they  should  answer  the  following  questions: 

•  What  effort  is  required  to  develop  all  the  documents  required? 

•  What  is  the  total  time  required  and  interim  milestones? 

•  What  supporting  documentation  (from  other  areas)  is  required  for  the  interim  milestones? 

•  What  supporting  technology  development/investigation  is  required? 

•  Can  an  overall  time-phased  implementation  provide  meaningful  interim  products? 

•  What  existing  work  is  available? 

•  What  can  a  “trial  run”  accomplish? 
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